Problem Statement¶
Business Context¶
The prices of the stocks of companies listed under a global exchange are influenced by a variety of factors, with the company's financial performance, innovations and collaborations, and market sentiment being factors that play a significant role. News and media reports can rapidly affect investor perceptions and, consequently, stock prices in the highly competitive financial industry. With the sheer volume of news and opinions from a wide variety of sources, investors and financial analysts often struggle to stay updated and accurately interpret its impact on the market. As a result, investment firms need sophisticated tools to analyze market sentiment and integrate this information into their investment strategies.
Problem Definition¶
With an ever-rising number of news articles and opinions, an investment startup aims to leverage artificial intelligence to address the challenge of interpreting stock-related news and its impact on stock prices. They have collected historical daily news for a specific company listed under NASDAQ, along with data on its daily stock price and trade volumes.
As a member of the Data Science and AI team in the startup, you have been tasked with analyzing the data, developing an AI-driven sentiment analysis system that will automatically process and analyze news articles to gauge market sentiment, and summarizing the news at a weekly level to enhance the accuracy of their stock price predictions and optimize investment strategies. This will empower their financial analysts with actionable insights, leading to more informed investment decisions and improved client outcomes.
Data Dictionary¶
Date: The date the news was releasedNews: The content of news articles that could potentially affect the company's stock priceOpen: The stock price (in \$) at the beginning of the dayHigh: The highest stock price (in \$) reached during the dayLow: The lowest stock price (in \$) reached during the dayClose: The adjusted stock price (in \$) at the end of the dayVolume: The number of shares traded during the dayLabel: The sentiment polarity of the news content- 1: positive
- 0: neutral
- -1: negative
Please read the instructions carefully before starting the project.¶
Note: If the free-tier GPU of Google Colab is not accessible (due to unavailability or exhaustion of daily limit or other reasons), the following steps can be taken:
Wait for 12-24 hours until the GPU is accessible again or the daily usage limits are reset.
Switch to a different Google account and resume working on the project from there.
Try using the CPU runtime:
- To use the CPU runtime, click on Runtime => Change runtime type => CPU => Save
- One can also click on the Continue without GPU option to switch to a CPU runtime (kindly refer to the snapshot below)
- The instructions for running the code on the CPU are provided in the respective sections of the notebook.
Installing and Importing Necessary Libraries¶
In [1]:
#included this line to elimate the gensim and numpy dependency and version issues
!pip install --upgrade pip -q
In [2]:
# installing the sentence-transformers and gensim libraries for word embeddings
!pip install -U sentence-transformers gensim transformers tqdm -q
restart the session and execute the above 2 lines to eliminate the numpy error dependencies.
In [3]:
# to read and manipulate the data
import pandas as pd
import numpy as np
pd.set_option('max_colwidth', None) # setting column to the maximum column width as per the data
# to visualise data
import matplotlib.pyplot as plt
import seaborn as sns
# to use regular expressions for manipulating text data
import re
# to load the natural language toolkit
import nltk
nltk.download('stopwords') # loading the stopwords
nltk.download('wordnet') # loading the wordnet module that is used in stemming
# to remove common stop words
from nltk.corpus import stopwords
# to perform stemming
from nltk.stem.porter import PorterStemmer
# To encode the target variable
from sklearn.preprocessing import LabelEncoder
# Patching scipy.linalg before importing gensim
import scipy.linalg # Import scipy.linalg
from numpy import triu # Import triu from numpy
scipy.linalg.triu = triu # Inject triu into scipy.linalg
# To import Word2Vec
from gensim.models import Word2Vec
import sklearn.metrics as metrics
# To tune the model
from sklearn.model_selection import GridSearchCV
# Converting the Stanford GloVe model vector format to word2vec
from gensim.scripts.glove2word2vec import glove2word2vec
from gensim.models import KeyedVectors
# Deep Learning library
import torch
# to load transformer models
from sentence_transformers import SentenceTransformer
# To split data into train and test sets
from sklearn.model_selection import train_test_split
# To build a Random Forest model
from sklearn.ensemble import RandomForestClassifier
# To compute metrics to evaluate the model
from sklearn.metrics import confusion_matrix, accuracy_score, f1_score, precision_score, recall_score, classification_report
Loading the dataset¶
In [4]:
# mounting Google Drive
from google.colab import drive
drive.mount('/content/drive')
In [5]:
# loading the dataset
df = pd.read_csv('/content/drive/My Drive/Colab Notebooks/NLP/Project/stock_news.csv')
In [6]:
data = df.copy()
Data Overview¶
In [7]:
data.head()
Out[7]:
In [8]:
# checking a stock news
data.loc[3, 'News']
Out[8]:
In [9]:
data.shape
Out[9]:
Observation : There are 349 rows and 8 columns.
In [10]:
data.info()
Observations:
- There are 6 numerical columns and 2 object columns.
- We need to convert Date column which is object type to date type
In [11]:
# changing the data type of Date column
data['Date'] = pd.to_datetime(data['Date'])
In [12]:
#verify Date column after converting to datetime type
data.info()
Observations:
- We see that the Date column is converted to datetime type.
- There are 6 numerical columns, 1 datetime column and 1 object column.
In [13]:
data.describe().T
Out[13]:
Observations:
- Open: Mean Open price is 46.22 dollars, min price is 37.56 dollars and max price is 66.81 dollars.
- High: Mean High price is 46.70 dollars, min price is 37.81 dollars and max price is 67.06 dollars.
- Low: Mean Low price is 45.74 dollars,min price is 37.30 dollars and max price is 65.86 dollars.
- Volume: Mean Volume is 128948236, min Volume is 45448000 and max Volume is 244439200.
In [14]:
#check for null values
data.isnull().sum()
Out[14]:
Observations:
- There are no null values.
In [15]:
#check for duplicates
data.duplicated().sum()
Out[15]:
Observations:
- There are no duplicates.
Exploratory Data Analysis¶
Univariate Analysis¶
- Distribution of individual variables
- Compute and check the distribution of the length of news content
Label
In [16]:
# function to create labeled barplots
def labeled_barplot(data, feature, perc=False, n=None):
"""
Barplot with percentage at the top
data: dataframe
feature: dataframe column
perc: whether to display percentages instead of count (default is False)
n: displays the top n category levels (default is None, i.e., display all levels)
"""
total = len(data[feature]) # length of the column
count = data[feature].nunique()
if n is None:
plt.figure(figsize=(count + 1, 5))
else:
plt.figure(figsize=(n + 1, 5))
plt.xticks(rotation=90, fontsize=15)
ax = sns.countplot(
data=data,
x=feature,
legend=False,
hue=feature,
palette="Paired",
order=data[feature].value_counts().index[:n].sort_values(),
)
for p in ax.patches:
if perc == True:
label = "{:.1f}%".format(
100 * p.get_height() / total
) # percentage of each class of the category
else:
label = p.get_height() # count of each level of the category
x = p.get_x() + p.get_width() / 2 # width of the plot
y = p.get_height() # height of the plot
ax.annotate(
label,
(x, y),
ha="center",
va="center",
size=12,
xytext=(0, 5),
textcoords="offset points",
) # annotate the percentage
plt.show() # show the plot
Distribution of Sentiments
In [17]:
labeled_barplot(data, "Label", perc=True)
Observations:
- Class 0 (Neutral): 48.7% is neutral sentiment and is the majority class
- Class -1(Negative): 28.4% is negative sentiment and is the second highest class.
Class 1(Positive): 22.9% is positive sentiment and is the class with the least count.
We see that the data is imbalanced.
Open Price
In [18]:
sns.histplot(data['Open'],kde=True)
plt.show()
Observations:
- Open price is right skewed distribution.
In [19]:
#calculate the mode
data['Open'].mode()
Out[19]:
Observations:
- Mean Open price(46.22) > Median Open price(45.97) > Mode Open price(43.57)
In [20]:
# Creating a time series DataFrame with 'Date' as index
timeseries_df = data.set_index('Date')
In [21]:
timeseries_df['Month'] = timeseries_df.index.month # Extract month from the index
sns.boxplot(x='Month', y='Open', data=timeseries_df) # Box plot of 'Close' prices by month
plt.show()
Observations:
- There are outliers in Month 1, 3 and 4.
- Month 2 Open price has the wide range of prices.
High Price
In [22]:
sns.histplot(data['High'],kde=True)
plt.show()
Observations:
- High stock price is right skewed distribution.
In [23]:
#calculate the mode of High stock price
data['High'].mode()
Out[23]:
Observations:
- Mean High price(46.70) > Median High price(46.02) > Mode High price(43.78)
In [24]:
timeseries_df['Month'] = timeseries_df.index.month # Extract month from the index
sns.boxplot(x='Month', y='High', data=timeseries_df) # Box plot of 'Close' prices by month
plt.show()
Observations:
- There are outliers in month 1, 3 and 4.
- Month 2 Open price has the wide range of prices.
Low Price
In [25]:
sns.histplot(data['Low'],kde=True)
plt.show()
Observations:
Low stock price is right skewed distribution.
In [26]:
#calculate the mode of Low stock price
data['Low'].mode()
Out[26]:
Observations:
- Mean Low Price(45.74)>Median Low Price(45.63)>Mode Low Price(43.22)
In [27]:
timeseries_df['Month'] = timeseries_df.index.month # Extract month from the index
sns.boxplot(x='Month', y='Low', data=timeseries_df) # Box plot of 'Close' prices by month
plt.show()
Observations:
- There are outliers in month 1, 3 and 4.
- Month 2 Low price has the wide range of prices.
Close Price
In [28]:
sns.histplot(data['Close'],kde=True)
plt.show()
Observations:
- Its a right skewed distribution.
In [29]:
#calculate the mode
data['Close'].mode()
Out[29]:
Observations:
- Mean Close price(44.92) > Median Close price(44.59) > Mode Close price(42.47)
In [30]:
timeseries_df['Month'] = timeseries_df.index.month # Extract month from the index
sns.boxplot(x='Month', y='Close', data=timeseries_df) # Box plot of 'Close' prices by month
plt.show()
Observations:
- There are outliers in all the months.
Volume
In [31]:
sns.histplot(data['Volume'],kde=True)
plt.show()
Observations:
- Volume is a right skewed distribution.
In [32]:
sns.boxplot(x=data['Volume'])
plt.show()
Observations:
- There is outlier.
Compute and check the distribution of the length of news content
In [33]:
#calculate the length of each news article
data['news_length'] = data['News'].apply(len)
In [34]:
#check the newly added column 'news_length'
data.head()
Out[34]:
Check the distribution
Histogram
In [35]:
sns.histplot(data['news_length'], kde=True)
plt.xlabel('news length')
plt.ylabel('Frequency')
plt.title('Distribution of news length')
plt.show()
Observations:
- Its a left skewed distribution.
In [36]:
print(data['news_length'].describe())
Observations:
- Mean news length is 311 characters. Min is 110 characters and max is 394 characters.
In [37]:
sns.boxplot(x=data['news_length'])
plt.xlabel('news length')
plt.title('Box Plot of news length')
plt.show()
Observations:
- There are outliers in news length.
Bivariate Analysis¶
- Correlation
- Sentiment Polarity vs Price
- Date vs Price
Note: The above points are listed to provide guidance on how to approach bivariate analysis. Analysis has to be done beyond the above listed points to get maximum scores.
In [38]:
sns.pairplot(data=data,hue='Label')
Out[38]:
Observations:
- We see a positive correlation between
- Open price and High price
- Open price and Low price
- Open price and Close price
- High Price and Low Price
- High Price and Close Price
- Low Price and Close Price
In [39]:
plt.figure(figsize=(15, 7))
numeric_df = data.select_dtypes(include=['number'])
sns.heatmap(numeric_df.corr(), annot=True, vmin=-1, vmax=1, fmt=".2f", cmap="Spectral")
plt.show()
Observations:
- We see the highest positive correlation of 1.0 between the following
- Open price and High price
- Open price and Low price
- Open price and Close price
- High Price and Low Price
- High Price and Close Price
- Low Price and Close Price
Relationship of numerical variables on target variable¶
- Sentiment Polarity vs Price
- Sentiment polarity vs Open
- Sentiment polarity vs High
- Sentiment polarity vs Low
- Sentiment polarity vs Close
- Sentiment Polarity vs Volume
In [40]:
### Function to plot distributions
def distribution_plot_wrt_target(data, predictor, target):
fig, axs = plt.subplots(2, 3, figsize=(12, 10))
target_uniq = data[target].unique()
axs[0, 0].set_title("Distribution of target for target=" + str(target_uniq[0]))
sns.histplot(
data=data[data[target] == target_uniq[0]],
x=predictor,
kde=True,
ax=axs[0, 0],
color="teal",
)
axs[0, 1].set_title("Distribution of target for target=" + str(target_uniq[1]))
sns.histplot(
data=data[data[target] == target_uniq[1]],
x=predictor,
kde=True,
ax=axs[0, 1],
color="orange",
)
axs[0, 2].set_title("Distribution of target for target=" + str(target_uniq[2]))
sns.histplot(
data=data[data[target] == target_uniq[2]],
x=predictor,
kde=True,
ax=axs[0, 2],
color="blue",
)
axs[1, 0].set_title("Boxplot w.r.t target")
#sns.boxplot(data=data, x=target, y=predictor, ax=axs[1, 0], palette="gist_rainbow")
sns.boxplot(data=data, x=target, y=predictor, hue=target, ax=axs[1, 0], palette="gist_rainbow", legend=False)
axs[1, 1].set_title("Boxplot (without outliers) w.r.t target")
sns.boxplot(
data=data,
x=target,
y=predictor,
hue=target,
ax=axs[1, 1],
showfliers=False,
palette="gist_rainbow",
legend=False
)
plt.tight_layout()
plt.show()
Sentimental Polarity(Label) vs Open Price
In [41]:
distribution_plot_wrt_target(data, 'Open', 'Label')
Observations:
- Distribution is left skewed for negative,neutral and positive sentiments.
- There are outliers for all the sentiments.
- Median Open price of neutral and positive is around 47 and negative sentiment is less which is around 43.
Sentimental Polarity(Label) vs High Price
In [42]:
distribution_plot_wrt_target(data, 'High', 'Label')
Observations:
- Distribution is left skewed for negative,neutral and positive sentiments.
- There are outliers for all the sentiments.
- Median High price of neutral and positive is around 47 and negative sentiment is less which is around 43.
Sentimental Polarity(Label) vs Low Price
In [43]:
distribution_plot_wrt_target(data, 'Low', 'Label')
Observations:
- Distribution is left skewed for negative,neutral and positive sentiments.
- There are outliers for all the sentiments.
- Median Open price of neutral and positive is around 47 and negative sentiment is less which is around 43.
Sentimental Polarity(Label) vs Close Price
In [44]:
distribution_plot_wrt_target(data, 'Close', 'Label')
Observations:
- Distribution is left skewed for negative,neutral and positive sentiments.
- There are outliers for all the sentiments.
- Median Close price of neutral and positive is around 45 and negative sentiment is less which is around 42.
Sentimental Polarity(Label) vs Volume
In [45]:
distribution_plot_wrt_target(data, 'Volume', 'Label')
Observations:
- Distribution is left skewed for negative,neutral and positive sentiments.
- There are outliers for all the sentiments.
- Median Volume of neutral sentiment is around 1.20x10^8, negative sentiment is around 1.125x10^8, positive sentiment is around 1.15x10^8.
Price vs Date
Plot Open,High,Low and Close Prices Vs Date
In [46]:
#build timeseries plot
# Create subplots
fig,axes = plt.subplots(4, 1, figsize=(10, 12), sharex=True) # 4 rows, 1 column
# Plot each price on a separate subplot
sns.lineplot(x=timeseries_df.index, y=timeseries_df['Open'], label='Open', ax=axes[0])
sns.lineplot(x=timeseries_df.index, y=timeseries_df['High'], label='High', ax=axes[1])
sns.lineplot(x=timeseries_df.index, y=timeseries_df['Low'], label='Low', ax=axes[2])
sns.lineplot(x=timeseries_df.index, y=timeseries_df['Close'], label='Close', ax=axes[3])
# Add labels and titles
for ax, price_type in zip(axes, ['Open', 'High', 'Low', 'Close']):
ax.set_ylabel(price_type + ' Price')
ax.legend()
ax.grid(True)
axes[3].set_xlabel('Date') # X-axis label only on the bottom subplot
fig.suptitle('Time Series Plots of Stock Prices', fontsize=16) # Overall title
plt.xticks(rotation=45, ha='right') # Rotate x-axis labels
plt.tight_layout() # Adjust spacing
plt.show() # Display the plot
Observations:
- The graph pattern of Open,High, Low and Close price looks very similar over the period of time.
- The stock price trend
- begins to rise up from begining of the month, peaks before the mid of the month, the drops low by mid of the month.It stays low from mid of the month to the beginning of the next month and the cycle continues.
- The above cycle is true for all stock prices - Open, High, Low and Close price.
- The stock price range across all the prices are very similar.
Volume vs Date
In [47]:
#plot Volume vs date
sns.lineplot(x='Date', y='Volume', data=timeseries_df)
plt.xlabel('Date')
plt.ylabel('Volume')
plt.title('Time Series Plot of Volume')
plt.grid(True)
plt.xticks(rotation=45, ha='right')
plt.show()
Observations:
- Highest volume traded is around 2019-02-1 and lowest volume traded is between 2019-03-01 and 2019-03-15.
Volume of stocks traded every month
In [48]:
#Volume of stocks traded every month
timeseries_df['Month'] = timeseries_df.index.month
monthly_volume = timeseries_df.groupby('Month')['Volume'].sum()
sns.lineplot(x= monthly_volume.index, y=monthly_volume.values)
plt.xlabel('Month')
plt.ylabel('Total Volume')
plt.title('Monthly Volume of Stocks Traded')
plt.show()
Observations:
- Volume decreases significantly after month 1,it drops to 0.50x10^10 by month 2, it then raises to 1.00x10^10 by month 3 and drops to 0.75x10^10 by month 4
Price, Volume vs date¶
Plot Price and Volume in y axis and date on x axis to see how the change in price and Volume over a period of time happens and if there is a pattern.
Open Price, Volume vs date
In [49]:
fig, ax1 = plt.subplots(figsize=(10, 6))
ax1.plot(timeseries_df.index, timeseries_df['Open'], color='blue', label='Open Price')
ax1.set_xlabel('Date')
ax1.set_ylabel('Open Price', color='blue')
ax1.tick_params('y', labelcolor='blue')
ax2 = ax1.twinx() # Create a secondary y-axis
ax2.plot(timeseries_df.index, timeseries_df['Volume'], color='red', label='Volume')
ax2.set_ylabel('Volume', color='red')
ax2.tick_params('y', labelcolor='red')
fig.tight_layout()
plt.title('Time Series Plot of Open Price and Volume')
plt.grid(True)
plt.xticks(rotation=45, ha='right')
plt.show()
Observations:
- Changing trend in Volume and open price is not correlated.
- Open Price and Volume peaks at different times.
High Price, Volume vs date
In [50]:
fig, ax1 = plt.subplots(figsize=(10, 6))
ax1.plot(timeseries_df.index, timeseries_df['High'], color='blue', label='High Price')
ax1.set_xlabel('Date')
ax1.set_ylabel('High Price', color='blue')
ax1.tick_params('y', labelcolor='blue')
ax2 = ax1.twinx() # Create a secondary y-axis
ax2.plot(timeseries_df.index, timeseries_df['Volume'], color='red', label='Volume')
ax2.set_ylabel('Volume', color='red')
ax2.tick_params('y', labelcolor='red')
fig.tight_layout()
plt.title('Time Series Plot of High Price and Volume')
plt.grid(True)
plt.xticks(rotation=45, ha='right')
plt.show()
Observations:
- Changing trend in Volume and High price is not correlated.
- Open Price and Volume peaks at different times.
Low Price, Volume vs date
In [51]:
fig, ax1 = plt.subplots(figsize=(10, 6))
ax1.plot(timeseries_df.index, timeseries_df['Low'], color='blue', label='Low Price')
ax1.set_xlabel('Date')
ax1.set_ylabel('Low Price', color='blue')
ax1.tick_params('y', labelcolor='blue')
ax2 = ax1.twinx() # Create a secondary y-axis
ax2.plot(timeseries_df.index, timeseries_df['Volume'], color='red', label='Volume')
ax2.set_ylabel('Volume', color='red')
ax2.tick_params('y', labelcolor='red')
fig.tight_layout()
plt.title('Time Series Plot of Low Price and Volume')
plt.grid(True)
plt.xticks(rotation=45, ha='right')
plt.show()
Observations:
- Changing trend in Volume and Low price is not correlated.
- Open Price and Volume peaks at different times.
Close Price, Volume vs date
In [52]:
fig, ax1 = plt.subplots(figsize=(10, 6))
ax1.plot(timeseries_df.index, timeseries_df['Close'], color='blue', label='Close Price')
ax1.set_xlabel('Date')
ax1.set_ylabel('Close Price', color='blue')
ax1.tick_params('y', labelcolor='blue')
ax2 = ax1.twinx() # Create a secondary y-axis
ax2.plot(timeseries_df.index, timeseries_df['Volume'], color='red', label='Volume')
ax2.set_ylabel('Volume', color='red')
ax2.tick_params('y', labelcolor='red')
fig.tight_layout()
plt.title('Time Series Plot of Close Price and Volume')
plt.grid(True)
plt.xticks(rotation=45, ha='right')
plt.show()
Observations:
- Changing trend in Volume and Close price is not correlated.
- Open Price and Volume peaks at different times.
EDA Summary¶
Univariate Analysis
- Class 0 (Neutral): 48.7% is neutral sentiment and is the majority class
- Class -1(Negative): 28.4% is negative sentiment and is the second highest class.
- Class 1(Positive): 22.9% is positive sentiment and is the class with the least count. We see that the data is imbalanced.
Multivariate Analysis
Sentimental Polarity(Label) vs Open Price
- Distribution is left skewed for negative,neutral and positive sentiments.
- There are outliers for all the sentiments.
- Median Open price of neutral and positive is around 47 and negative sentiment is less which is around 43.
Sentimental Polarity(Label) vs High Price
- Distribution is left skewed for negative,neutral and positive sentiments.
- There are outliers for all the sentiments.
- Median High price of neutral and positive is around 47 and negative sentiment is less which is around 43.
Sentimental Polarity(Label) vs Low Price
- Distribution is left skewed for negative,neutral and positive sentiments.
- There are outliers for all the sentiments.
- Median Open price of neutral and positive is around 47 and negative sentiment is less which is around 43.
Sentimental Polarity(Label) vs Close Price
- Distribution is left skewed for negative,neutral and positive sentiments.
- There are outliers for all the sentiments.
- Median Close price of neutral and positive is around 45 and negative sentiment is less which is around 42.
Sentimental Polarity(Label) vs Volume
- Distribution is left skewed for negative,neutral and positive sentiments.
- There are outliers for all the sentiments.
- Median Volume of neutral sentiment is around 1.20x10^8, negative sentiment is around 1.125x10^8, positive sentiment is around 1.15x10^8.
Price vs Date
- The graph pattern of Open,High, Low and Close price looks very similar over the period of time.
- The stock price trend
- begins to rise up from begining of the month, peaks before the mid of the month, the drops low by mid of the month.It stays low from mid of the month to the beginning of the next month and the cycle continues.
- The above cycle is true for all stock prices - Open, High, Low and Close price.
- The stock price range across all the prices are very similar.
Volume of stocks traded every month
- Volume decreases significantly after month 1,it drops to 0.50x10^10 by month 2, it then raises to 1.00x10^10 by month 3 and drops to 0.75x10^10 by month 4.
Data Preprocessing¶
In [53]:
dataset = data.copy()
Preprocessing the textual column¶
In [54]:
# Loading the Porter Stemmer
ps = PorterStemmer()
In [55]:
def preprocess_text(text):
# Convert to lowercase
text = text.lower()
# Remove special characters and numbers
text = re.sub(r'[^A-Za-z\s]', '', text)
# Remove extra whitespaces
text = re.sub(r'\s+', ' ', text).strip()
# Split text into separate words
words = text.split()
# Removing English language stopwords
text = ' '.join([word for word in words if word not in stopwords.words('english')])
# Applying the Porter Stemmer on every word of a message and joining the stemmed words back into a single string
text = ' '.join([ps.stem(word) for word in words])
return text
In [56]:
# preprocessing the textual column
dataset['News_clean'] = dataset['News'].apply(preprocess_text)
In [57]:
#display cleaned text
dataset.head()
Out[57]:
Please Note : Moved splitting the dataset after word embeddings
Word Embeddings¶
Word2Vec¶
In [58]:
# Creating a list of all words in our data
words_list = [item.split(" ") for item in dataset['News_clean'].values]
In [59]:
# Creating an instance of Word2Vec
vec_size = 300
model_W2V = Word2Vec(words_list, vector_size = vec_size, min_count = 1, window=5, workers = 6)
In [60]:
# Checking the size of the vocabulary
print("Length of the vocabulary is", len(list(model_W2V.wv.key_to_index)))
Let's check out a few word embeddings obtained using the model
In [61]:
# Checking the word embedding of a random word
word = "market"
model_W2V.wv[word]
Out[61]:
In [62]:
# Checking the word embedding of a random word
word = "stock"
model_W2V.wv[word]
Out[62]:
In [63]:
# Checking the word embedding of a random word
word = "analyst"
model_W2V.wv[word]
Out[63]:
In [64]:
# Retrieving the words present in the Word2Vec model's vocabulary
words = list(model_W2V.wv.key_to_index.keys())
# Retrieving word vectors for all the words present in the model's vocabulary
wvs = model_W2V.wv[words].tolist()
# Creating a dictionary of words and their corresponding vectors
word_vector_dict = dict(zip(words, wvs))
In [65]:
def average_vectorizer_Word2Vec(doc):
# Initializing a feature vector for the sentence
feature_vector = np.zeros((vec_size,), dtype="float64")
# Creating a list of words in the sentence that are present in the model vocabulary
words_in_vocab = [word for word in doc.split() if word in words]
# adding the vector representations of the words
for word in words_in_vocab:
feature_vector += np.array(word_vector_dict[word])
# Dividing by the number of words to get the average vector
if len(words_in_vocab) != 0:
feature_vector /= len(words_in_vocab)
return feature_vector
In [66]:
# creating a dataframe of the vectorized documents
df_Word2Vec = pd.DataFrame(dataset['News_clean'].apply(average_vectorizer_Word2Vec).tolist(), columns=['Feature '+str(i) for i in range(vec_size)])
df_Word2Vec
Out[66]:
Glove¶
In [67]:
from gensim.models import KeyedVectors
# load the Stanford GloVe model
filename = '/content/drive/MyDrive/Colab Notebooks/NLP/Project/glove.6B.100d.txt.word2vec'
glove_model = KeyedVectors.load_word2vec_format(filename, binary=False)
In [68]:
# Checking the size of the vocabulary
print("Length of the vocabulary is", len(glove_model.index_to_key))
In [69]:
# Checking the word embedding of a random word
word = "market"
glove_model[word]
Out[69]:
In [70]:
# Checking the word embedding of a random word
word = "stock"
glove_model[word]
Out[70]:
In [71]:
# Checking the word embedding of a random word
word = "analyst"
glove_model[word]
Out[71]:
In [72]:
# Retrieving the words present in the GloVe model's vocabulary
glove_words = glove_model.index_to_key
# Creating a dictionary of words and their corresponding vectors
glove_word_vector_dict = dict(zip(glove_model.index_to_key,list(glove_model.vectors)))
In [73]:
vec_size=100
In [74]:
def average_vectorizer_GloVe(doc):
# Initializing a feature vector for the sentence
feature_vector = np.zeros((vec_size,), dtype="float64")
# Creating a list of words in the sentence that are present in the model vocabulary
words_in_vocab = [word for word in doc.split() if word in glove_words]
# adding the vector representations of the words
for word in words_in_vocab:
feature_vector += np.array(glove_word_vector_dict[word])
# Dividing by the number of words to get the average vector
if len(words_in_vocab) != 0:
feature_vector /= len(words_in_vocab)
return feature_vector
In [75]:
# creating a dataframe of the vectorized documents
df_Glove = pd.DataFrame(dataset['News_clean'].apply(average_vectorizer_GloVe).tolist(), columns=['Feature '+str(i) for i in range(vec_size)])
df_Glove
Out[75]:
Sentence Transformer¶
In [76]:
# defining the model
sent_model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
In [77]:
# setting the device to GPU if available, else CPU
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
In [78]:
# encoding the dataset
embedding_matrix = sent_model.encode(dataset['News'], device=device, show_progress_bar=True)
In [79]:
# printing the shape of the embedding matrix
embedding_matrix.shape
Out[79]:
In [80]:
# printing the embedding vector of the first news in the dataset
embedding_matrix[0,:]
Out[80]:
Data Pre-processing¶
Splitting the dataset
In [81]:
# Creating dependent and independent variables
X_word2vec = df_Word2Vec.copy()
X_glove = df_Glove.copy()
X_sent_transformer = embedding_matrix.copy()
y=dataset['Label']
In [82]:
def split(X,y):
# Initial split into training (80%) and testing (20%)
X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.20, stratify=y, random_state=42)
# Further split the temporary set into validation (10%) and test (10%) sets
X_valid, X_test, y_valid, y_test = train_test_split(X_temp, y_temp, test_size=0.50, stratify=y_temp, random_state=42)
return X_train,X_valid,X_test,y_train,y_valid,y_test
In [83]:
#Splitting the dataset.
X_train_word2vec,X_valid_word2vec,X_test_word2vec,y_train_word2vec,y_valid_word2vec,y_test_word2vec=split(X_word2vec,y)
X_train_glove,X_valid_glove,X_test_glove,y_train_glove,y_valid_glove,y_test_glove=split(X_glove,y)
X_train_sent_transformer,X_valid_sent_transformer,X_test_sent_transformer,y_train_sent_transformer,y_valid_sent_transformer,y_test_sent_transformer=split(X_sent_transformer,y)
Check the shape of training,validation and testing datasets of Word2vec,Glove and Sentence Transformers
In [84]:
print(X_train_word2vec.shape, X_test_word2vec.shape, X_valid_word2vec.shape)
In [85]:
print(y_train_word2vec.shape, y_test_word2vec.shape, y_valid_word2vec.shape)
In [86]:
print(X_train_glove.shape, X_test_glove.shape, X_valid_glove.shape)
In [87]:
print(y_train_glove.shape, y_test_glove.shape, y_valid_glove.shape)
In [88]:
print(X_train_sent_transformer.shape, X_test_sent_transformer.shape, X_valid_sent_transformer.shape)
In [89]:
print(y_train_sent_transformer.shape, y_test_sent_transformer.shape, y_valid_sent_transformer.shape)
Observations:
- Word2vec datasets
- Training: 279 rows and each sample is represented by 300 dimensional vector(word2vec embeddings)
- Validation: validation data has 35 samples, with each sample having a 300-dimensional Word2Vec embedding
- Testing: Testing data has 35 samples, with each sample having a 300-dimensional Word2Vec embedding
- Glove datasets
- Training: 279 rows and each sample is represented by 100 dimensional vector(Glove embeddings)
- Validation: validation data has 35 samples, with each sample having a 100-dimensional Word2Vec embedding
- Testing: Testing data has 35 samples, with each sample having a 100-dimensional Word2Vec embedding
- Sentence Transformer datasets
- Training: 279 rows and each sample is represented by 300 dimensional vector(Sentence transformer embeddings)
- Validation: validation data has 35 samples, with each sample having a 300-dimensional Word2Vec embedding
- Testing: Testing data has 35 samples, with each sample having a 300-dimensional Word2Vec embedding
Sentiment Analysis¶
Model Evaluation Criterion
Model can make wrong prediction:
- Model predicting a possitive sentiment when the actual outcome is negative - FALSE POSITIVES
- Model predicting a negative sentiment when the actual outcome is positive - FALSE NEGATIVE
Which case is more important?
- Predicting a negative sentiment when the actual outcome is positive could result in missing out on opportunities to buy stocks that would have gained value.
- There is data imbalance as well in which Recall becomes more critical.
How to reduce this loss?
- Maximize Recall to reduce False Negatives.
Model Building¶
Function for confusion matrix
In [90]:
def confusion_matrix_sklearn(model, predictors, target):
"""
To plot the confusion_matrix with percentages
model: classifier
predictors: independent variables
target: dependent variable
"""
y_pred = model.predict(predictors)
cm = confusion_matrix(target, y_pred)
labels = np.asarray(
[
["{0:0.0f}".format(item) + "\n{0:.2%}".format(item / cm.flatten().sum())]
for item in cm.flatten()
]
).reshape(cm.shape[0], cm.shape[1])
plt.figure(figsize=(6, 4))
sns.heatmap(cm, annot=labels, fmt="")
plt.ylabel("True label")
plt.xlabel("Predicted label")
Random Forest Model (default with Word2Vec)¶
In [91]:
# Building the model
rf_word2vec = RandomForestClassifier(random_state = 42)
# Fitting on train data
rf_word2vec.fit(X_train_word2vec, y_train_word2vec)
Out[91]:
Confusion Matrix
In [92]:
confusion_matrix_sklearn(rf_word2vec, X_train_word2vec, y_train_word2vec)
Observations:
- We see that all predicted values matches the true/actual values and it looks like that the data is overfitting.
In [93]:
confusion_matrix_sklearn(rf_word2vec, X_valid_word2vec, y_valid_word2vec)
Observations:
- Around 40% of the predicted values match with the actual values.
In [94]:
# Predicting on train data
y_pred_train_word2vec = rf_word2vec.predict(X_train_word2vec)
# Predicting on validation data
y_pred_valid_word2vec = rf_word2vec.predict(X_valid_word2vec)
Classification Report
In [95]:
print(classification_report(y_train_word2vec, y_pred_train_word2vec))
Observations:
- For training data, We have a Recall score of 100% which indicates that the model is overfitting.
In [96]:
default_word2vec_report = classification_report(y_valid_word2vec, y_pred_valid_word2vec)
print(default_word2vec_report)
Observations:
- We got a weighted recall score of 40% which indicates that the model was overfitting.
- Negative sentiment recall score is 10%
- Neutral sentiment recall score is 71%
- Positive sentiment recall score is 12%
Random Forest Model (default with GloVe)¶
In [97]:
# Building the model
rf_glove = RandomForestClassifier(random_state = 42)
# Fitting on train data
rf_glove.fit(X_train_glove, y_train_glove)
Out[97]:
Confusion Matrix
In [98]:
confusion_matrix_sklearn(rf_glove, X_train_glove, y_train_glove)
Observations:
- We see that all predicted values matches the true/actual values and it looks like that the data is overfitting.
In [99]:
confusion_matrix_sklearn(rf_glove, X_valid_glove, y_valid_glove)
Observations:
- Around 46% of the predicted values match the true values.
In [100]:
# Predicting on train data
y_pred_train_glove = rf_glove.predict(X_train_glove)
# Predicting on validation data
y_pred_valid_glove = rf_glove.predict(X_valid_glove)
Classification Report
In [101]:
print(classification_report(y_train_glove, y_pred_train_glove))
Observations:
- For training data, We have a Recall score of 100% which indicates that the model is overfitting.
In [102]:
default_glove_report = classification_report(y_valid_glove, y_pred_valid_glove)
print(default_glove_report)
Observations:
- We got a weighted recall score of 46% which indicates that the model was overfitting.
- Negative sentiment recall score is 20%
- Neutral sentiment recall score is 76%
- Positive sentiment recall score is 12%
Random Forest Model (default with Sentence Transformer)¶
In [103]:
rf_sent_transformer = RandomForestClassifier(random_state = 42)
# Fitting on train data
rf_sent_transformer.fit(X_train_sent_transformer, y_train_sent_transformer)
Out[103]:
Confusion Matrix
In [104]:
confusion_matrix_sklearn(rf_sent_transformer, X_train_sent_transformer, y_train_sent_transformer)
Observations:
- We see that all predicted values matches the true/actual values and it looks like that the data is overfitting.
In [105]:
confusion_matrix_sklearn(rf_sent_transformer, X_valid_sent_transformer, y_valid_sent_transformer)
Observations:
- For Validation dataset, around 43% of the predicted values match the true values.
In [106]:
# Predicting on train data
y_pred_train_sent_transformer = rf_sent_transformer.predict(X_train_sent_transformer)
# Predicting on validation data
y_pred_valid_sent_transformer = rf_sent_transformer.predict(X_valid_sent_transformer)
Classification Report
In [107]:
print(classification_report(y_train_sent_transformer, y_pred_train_sent_transformer))
Observations:
- For training data, We have a Recall score of 100% which indicates that the model is overfitting.
In [108]:
#included zero_division=1 to address the warning related to precision values
default_sent_report = classification_report(y_valid_sent_transformer, y_pred_valid_sent_transformer, zero_division=1)
print(default_sent_report)
Observations:
- We got a weighted recall score of 43% which indicates that the model was overfitting.
- Negative sentiment recall score is 10%
- Neutral sentiment recall score is 76%
- Positive sentiment recall score is 12%
We'll try to address the class imbalance problem now with Class weights.
Random Forest (with class_weights and Word2Vec)¶
In [109]:
rf_word2vec_balanced = RandomForestClassifier(class_weight="balanced", random_state=42)
rf_word2vec_balanced.fit(X_train_word2vec, y_train_word2vec)
Out[109]:
Confusion Matrix
In [110]:
confusion_matrix_sklearn(rf_word2vec_balanced, X_train_word2vec, y_train_word2vec)
Observations:
- We see that all predicted values matches the true/actual values and it looks like that the data is overfitting.
In [111]:
confusion_matrix_sklearn(rf_word2vec_balanced, X_valid_word2vec, y_valid_word2vec)
Observations:
- For Validation dataset, around 43% of the predicted values match the true values.
In [112]:
# Predicting on train data
y_pred_train_word2vec_balanced = rf_word2vec_balanced.predict(X_train_word2vec)
# Predicting on test data
y_pred_valid_word2vec_balanced = rf_word2vec_balanced.predict(X_valid_word2vec)
Classification report
In [113]:
print(classification_report(y_train_word2vec, y_pred_train_word2vec_balanced))
Observations:
- For training data, We have a Recall score of 100% which indicates that the model is overfitting.
In [114]:
weighted_word2vec_report = classification_report(y_valid_word2vec, y_pred_valid_word2vec_balanced)
print(weighted_word2vec_report)
Observations:
- We got a weighted recall score of 43% which indicates that the model was overfitting.
- Negative sentiment recall score is 10%
- Neutral sentiment recall score is 76%
- Positive sentiment recall score is 12%
Random Forest (with class_weights and GloVe)¶
In [115]:
rf_glove_balanced = RandomForestClassifier(class_weight="balanced", random_state=42)
rf_glove_balanced.fit(X_train_glove, y_train_glove)
Out[115]:
Confusion Matrix
In [116]:
confusion_matrix_sklearn(rf_glove_balanced, X_train_glove, y_train_glove)
Observations:
- We see that all predicted values matches the true/actual values and it looks like that the data is overfitting.
In [117]:
confusion_matrix_sklearn(rf_glove_balanced, X_valid_glove, y_valid_glove)
Observations,
- For validation dataset, we see that around 40% of the predicted values match with the true values.
In [118]:
# Predicting on train data
y_pred_train_glove_balanced = rf_glove_balanced.predict(X_train_glove)
# Predicting on test data
y_pred_valid_glove_balanced = rf_glove_balanced.predict(X_valid_glove)
Classification report
In [119]:
print(classification_report(y_train_glove, y_pred_train_glove_balanced))
Observations:
- For training data, We have a Recall score of 100% which indicates that the model is overfitting.
In [120]:
weighted_glove_report = classification_report(y_valid_glove, y_pred_valid_glove_balanced, zero_division=1)
print(weighted_glove_report)
Observations:
- We got a weighted recall score of 40% which indicates that the model was overfitting.
- Negative sentiment recall score is 10%
- Neutral sentiment recall score is 76%
- Positive sentiment recall score is 0%
Random Forest (with class_weights and Sentence Transformer)¶
In [121]:
rf_sent_transformer_balanced = RandomForestClassifier(class_weight="balanced", random_state=42)
rf_sent_transformer_balanced.fit(X_train_sent_transformer, y_train_sent_transformer)
Out[121]:
Confusion Matrix
In [122]:
confusion_matrix_sklearn(rf_sent_transformer_balanced, X_train_sent_transformer, y_train_sent_transformer)
Observations:
- We see that all predicted values matches the true/actual values and it looks like that the data is overfitting.
In [123]:
confusion_matrix_sklearn(rf_sent_transformer_balanced, X_valid_sent_transformer, y_valid_sent_transformer)
Observations:
- For validation dataset, we see around 46% of the predicted values match the actual values.
In [124]:
#predicting on train data
y_pred_train_sent_transformer_balanced = rf_sent_transformer_balanced.predict(X_train_sent_transformer)
#predicting on test data
y_pred_valid_sent_transformer_balanced = rf_sent_transformer_balanced.predict(X_valid_sent_transformer)
Classification Report
In [125]:
print(classification_report(y_train_sent_transformer, y_pred_train_sent_transformer_balanced))
Observations:
- For training data, We have a Recall score of 100% which indicates that the model is overfitting.
In [126]:
weighted_sent_report = classification_report(y_valid_sent_transformer, y_pred_valid_sent_transformer_balanced, zero_division=1)
print(weighted_sent_report)
Observations:
- We got a weighted recall score of 46% which indicates that the model was overfitting.
- Negative sentiment recall score is 10%
- Neutral sentiment recall score is 82%
- Positive sentiment recall score is 12%
Random Forest (with hyperparameter tuning and Word2Vec)¶
In [127]:
import sklearn.metrics as metrics
from sklearn.model_selection import GridSearchCV
# Choose the type of classifier
rf_tuned = RandomForestClassifier(class_weight="balanced", random_state=42)
# defining the hyperparameter grid for tuning
parameters = {
"max_depth": list(np.arange(4, 15, 2)),
"max_features": ["sqrt", 0.5, 0.7],
"min_samples_split": [5, 6, 7],
"n_estimators": np.arange(30, 110, 10),
}
# defining the type of scoring used to compare parameter combinations
# we need to specify the mechanism of averaging as we have more than 2 target classes
scorer = metrics.make_scorer(metrics.recall_score, average='weighted')
# running the grid search
grid_obj = GridSearchCV(rf_tuned, parameters, scoring=scorer, cv=3, n_jobs=-1)
grid_obj = grid_obj.fit(X_train_word2vec, y_train_word2vec)
In [128]:
# Creating a new model with the best combination of parameters
rf_word2vec_tuned = grid_obj.best_estimator_
# Fit the new model to the data
rf_word2vec_tuned.fit(X_train_word2vec, y_train_word2vec)
Out[128]:
Confusion Matrix
In [129]:
confusion_matrix_sklearn(rf_word2vec_tuned, X_train_word2vec, y_train_word2vec)
Observations:
- We see that all predicted values matches the true/actual values and it looks like that the data is overfitting.
In [130]:
confusion_matrix_sklearn(rf_word2vec_tuned, X_valid_word2vec, y_valid_word2vec)
Observations:
- For validation dataset, around 40% of the predicted values match the actual values.
In [131]:
# Predicting on train data
y_pred_train_word2vec_tuned = rf_word2vec_tuned.predict(X_train_word2vec)
# Predicting on validation data
y_pred_valid_word2vec_tuned = rf_word2vec_tuned.predict(X_valid_word2vec)
Classification Report
In [132]:
print(classification_report(y_train_word2vec, y_pred_train_word2vec_tuned))
Observations:
- For training data, We have a Recall score of 99% which still indicates that the model is overfitting.
In [133]:
tuned_word2vec_report = classification_report(y_valid_word2vec, y_pred_valid_word2vec_tuned)
print(tuned_word2vec_report)
Observations:
- We got a weighted recall score of 40% which indicates that the model was overfitting.
- Negative sentiment recall score is 10%
- Neutral sentiment recall score is 65%
- Positive sentiment recall score is 25%
Random Forest (with hyperparametertuning and GloVe)¶
In [134]:
# Choose the type of classifier
rf_tuned = RandomForestClassifier(class_weight="balanced", random_state=42)
# defining the hyperparameter grid for tuning
parameters = {
"max_depth": list(np.arange(4, 15, 2)),
"max_features": ["sqrt", 0.5, 0.7],
"min_samples_split": [5, 6, 7],
"n_estimators": np.arange(30, 110, 10),
}
# defining the type of scoring used to compare parameter combinations
# we need to specify the mechanism of averaging as we have more than 2 target classes
scorer = metrics.make_scorer(metrics.recall_score, average='weighted')
# running the grid search
grid_obj = GridSearchCV(rf_tuned, parameters, scoring=scorer, cv=3, n_jobs=-1)
grid_obj = grid_obj.fit(X_train_glove, y_train_glove)
In [135]:
# Creating a new model with the best combination of parameters
rf_glove_tuned = grid_obj.best_estimator_
# Fit the new model to the data
rf_glove_tuned.fit(X_train_glove, y_train_glove)
Out[135]:
Confusion Matrix
In [136]:
#Printing the confusion matrix
confusion_matrix_sklearn(rf_glove_tuned, X_train_glove, y_train_glove)
Observations:
- We see that all predicted values matches the true/actual values and it looks like that the data is overfitting.
In [137]:
#Printing the confusion matrix
confusion_matrix_sklearn(rf_glove_tuned, X_valid_glove, y_valid_glove)
Observations:
- For validation dataset, we see that around 37% of the predicted values match the actual values.
In [138]:
# Predicting on train data
y_pred_train_glove_tuned = rf_glove_tuned.predict(X_train_glove)
# Predicting on validation data
y_pred_valid_glove_tuned = rf_glove_tuned.predict(X_valid_glove)
Classification Report
In [139]:
print(classification_report(y_train_glove, y_pred_train_glove_tuned))
Observations:
- For training data, We have a Recall score of 100% which indicates that the model is overfitting.
In [140]:
tuned_glove_report = classification_report(y_valid_glove, y_pred_valid_glove_tuned)
print(tuned_glove_report)
Observations:
- We got a weighted recall score of 37% which indicates that the model was overfitting.
- Negative sentiment recall score is 10%
- Neutral sentiment recall score is 65%
- Positive sentiment recall score is 12%
Random Forest (with hyperparameter tuning and Sentence transformer)¶
In [141]:
# Choose the type of classifier
rf_tuned = RandomForestClassifier(class_weight="balanced", random_state=42)
# defining the hyperparameter grid for tuning
parameters = {
"max_depth": list(np.arange(4, 15, 2)),
"max_features": ["sqrt", 0.5, 0.7],
"min_samples_split": [5, 6, 7],
"n_estimators": np.arange(30, 110, 10),
}
# defining the type of scoring used to compare parameter combinations
# we need to specify the mechanism of averaging as we have more than 2 target classes
scorer = metrics.make_scorer(metrics.recall_score, average='weighted')
# running the grid search
grid_obj = GridSearchCV(rf_tuned, parameters, scoring=scorer, cv=3, n_jobs=-1)
grid_obj = grid_obj.fit(X_train_sent_transformer, y_train_sent_transformer)
In [142]:
# Creating a new model with the best combination of parameters
rf_sent_tuned = grid_obj.best_estimator_
# Fit the new model to the data
rf_sent_tuned.fit(X_train_sent_transformer, y_train_sent_transformer)
Out[142]:
In [143]:
#Printing the confusion matrix
confusion_matrix_sklearn(rf_sent_tuned, X_train_sent_transformer, y_train_sent_transformer)
Observations:
- We see that all predicted values matches the true/actual values and it looks like that the data is overfitting.
In [144]:
#Printing the confusion matrix
confusion_matrix_sklearn(rf_sent_tuned, X_valid_sent_transformer, y_valid_sent_transformer)
Observations:
- For validation dataset, we see that around 43% of the predicted values match the actual values.
In [145]:
# Predicting on train data
y_pred_train_sent_tuned = rf_sent_tuned.predict(X_train_sent_transformer)
# Predicting on validation data
y_pred_valid_sent_tuned = rf_sent_tuned.predict(X_valid_sent_transformer)
Classification Report
In [146]:
print(classification_report(y_train_sent_transformer, y_pred_train_sent_tuned))
Observations:
- For training data, We have a Recall score of 97%.
In [147]:
tuned_sent_report = classification_report(y_valid_sent_transformer, y_pred_valid_sent_tuned)
print(tuned_sent_report)
Observations:
- We got a weighted recall score of 43% which indicates that the model was overfitting.
- Negative sentiment recall score is 20%
- Neutral sentiment recall score is 65%
- Positive sentiment recall score is 25%
XGBoost (default and Word2Vec)¶
In [148]:
import pandas as pd
from xgboost import XGBClassifier
#XGBoost doesnt take negative values/classes in y. So map the classes to positive number
# Assuming y_train_word2vec is a pandas Series
y_train_xgb_word2vec = y_train_word2vec.map({-1: 0, 0: 1, 1: 2}) # Change -1 to 0, 0 to 1, and 1 to 2
#Fitting the model
xgb_word2vec = XGBClassifier(random_state=42, eval_metric='logloss')
xgb_word2vec.fit(X_train_word2vec, y_train_xgb_word2vec)
y_valid_xgb_word2vec = y_valid_word2vec.map({-1: 0, 0: 1, 1: 2}) # Change -1 to 0, 0 to 1, and 1 to 2
Confusion Matrix
In [149]:
confusion_matrix_sklearn(xgb_word2vec, X_train_word2vec, y_train_xgb_word2vec)
Observations:
- We see that all predicted values matches the true/actual values and it looks like that the data is overfitting.
In [150]:
confusion_matrix_sklearn(xgb_word2vec, X_valid_word2vec, y_valid_xgb_word2vec)
Observations:
- For Validation dataset, we see that around 34% of the predicted values match the actual values.
In [151]:
# Predicting on train data
y_pred_train_xgbword2vec = xgb_word2vec.predict(X_train_word2vec)
# Predicting on validation data
y_pred_valid_xgbword2vec = xgb_word2vec.predict(X_valid_word2vec)
Classification Report
In [152]:
print(classification_report(y_train_xgb_word2vec, y_pred_train_xgbword2vec))
Observations:
- For training data, We have a Recall score of 100% which indicates that the model is overfitting.
In [153]:
default_xgb_word2vec_report = classification_report(y_valid_xgb_word2vec, y_pred_valid_xgbword2vec)
print(default_xgb_word2vec_report)
Observations:
- We got a weighted recall score of 34% which indicates that the model was overfitting.
- Negative sentiment recall score is 0%
- Neutral sentiment recall score is 65%
- Positive sentiment recall score is 0%
XGBoost Model (default with GloVe)¶
In [154]:
#XGBoost doesnt take negative values/classes in y. So map the classes to positive number
y_train_xgb_glove = y_train_glove.map({-1: 0, 0: 1, 1: 2}) # Change -1 to 0, 0 to 1, and 1 to 2
y_valid_xgb_glove = y_valid_glove.map({-1: 0, 0: 1, 1: 2}) # Change -1 to 0, 0 to 1, and 1 to 2
In [155]:
# Building the model
xgb_glove = XGBClassifier(random_state = 42)
# Fitting on train data
xgb_glove.fit(X_train_glove, y_train_xgb_glove)
Out[155]:
Confusion Matrix
In [156]:
confusion_matrix_sklearn(xgb_glove, X_train_glove, y_train_xgb_glove)
Observations:
- We see that all predicted values matches the true/actual values and it looks like that the data is overfitting.
In [157]:
confusion_matrix_sklearn(xgb_glove, X_valid_glove, y_valid_xgb_glove)
Observations:
- For Validation dataset, we see that around 40% of the predicted values match the actual values.
In [158]:
# Predicting on train data
y_pred_train_xgb_glove = xgb_glove.predict(X_train_glove)
# Predicting on validation data
y_pred_valid_xgb_glove = xgb_glove.predict(X_valid_glove)
Classification Report
In [159]:
print(classification_report(y_train_xgb_glove, y_pred_train_xgb_glove))
Observations:
- For training data, We have a Recall score of 100% which indicates that the model is overfitting.
In [160]:
default_xgb_glove_report = classification_report(y_valid_xgb_glove, y_pred_valid_xgb_glove)
print(default_xgb_glove_report)
Observations:
- We got a weighted recall score of 40% which indicates that the model was overfitting.
- Negative sentiment recall score is 20%
- Neutral sentiment recall score is 65%
- Positive sentiment recall score is 12%
XGBoost Model (default with Sentence Transformer)¶
In [161]:
#XGBoost doesnt take negative values/classes in y. So map the classes to positive number
y_train_xgb_sent_transformer = y_train_sent_transformer.map({-1: 0, 0: 1, 1: 2}) # Change -1 to 0, 0 to 1, and 1 to 2
y_valid_xgb_sent_transformer = y_valid_sent_transformer.map({-1: 0, 0: 1, 1: 2}) # Change -1 to 0, 0 to 1, and 1 to 2
In [162]:
xgb_sent_transformer = XGBClassifier(random_state = 42)
# Fitting on train data
xgb_sent_transformer.fit(X_train_sent_transformer, y_train_xgb_sent_transformer)
Out[162]:
Confusion Matrix
In [163]:
confusion_matrix_sklearn(xgb_sent_transformer, X_train_sent_transformer, y_train_xgb_sent_transformer)
Observations:
- We see that all predicted values matches the true/actual values and it looks like that the data is overfitting.
In [164]:
confusion_matrix_sklearn(xgb_sent_transformer, X_valid_sent_transformer, y_valid_xgb_sent_transformer)
Observations:
- For validation dataset, we see that around 46% of the predicted values match the actual values.
In [165]:
# Predicting on train data
y_pred_train_xgb_sent_transformer = xgb_sent_transformer.predict(X_train_sent_transformer)
# Predicting on validation data
y_pred_valid_xgb_sent_transformer = xgb_sent_transformer.predict(X_valid_sent_transformer)
Classification Report
In [166]:
print(classification_report(y_train_xgb_sent_transformer, y_pred_train_xgb_sent_transformer))
Observations:
- For training data, We have a Recall score of 100% which indicates that the model is overfitting.
In [167]:
#included zero_division=1 to address the warning related to precision values
default_xgb_sent_report = classification_report(y_valid_xgb_sent_transformer, y_pred_valid_xgb_sent_transformer, zero_division=1)
print(default_xgb_sent_report)
Observations:
- We got a weighted recall score of 46% which indicates that the model was overfitting.
- Negative sentiment recall score is 30%
- Neutral sentiment recall score is 59%
- Positive sentiment recall score is 38%
XGBoost (with hyperparameter tuning and Word2Vec)¶
In [168]:
import sklearn.metrics as metrics
from sklearn.model_selection import GridSearchCV
# Choose the type of classifier
xgb_tuned = XGBClassifier(random_state=42, eval_metric='mlogloss')
# defining the hyperparameter grid for tuning
parameters = {
"n_estimators": [10,30,50],
"subsample":[0.7,0.9,1],
"learning_rate":[0.05, 0.1,0.2],
"colsample_bytree":[0.7,0.9,1],
"colsample_bylevel":[0.5,0.7,1]
}
# defining the type of scoring used to compare parameter combinations
# we need to specify the mechanism of averaging as we have more than 2 target classes
scorer = metrics.make_scorer(metrics.recall_score, average='weighted')
# running the grid search
grid_obj = GridSearchCV(xgb_tuned, parameters, scoring=scorer, cv=3, n_jobs=-1)
grid_obj = grid_obj.fit(X_train_word2vec, y_train_xgb_word2vec)
In [169]:
# Creating a new model with the best combination of parameters
xgb_word2vec_tuned = grid_obj.best_estimator_
# Fit the new model to the data
xgb_word2vec_tuned.fit(X_train_word2vec, y_train_xgb_word2vec)
Out[169]:
Confusion Matrix
In [170]:
confusion_matrix_sklearn(xgb_word2vec_tuned, X_train_word2vec, y_train_xgb_word2vec)
Observations:
- Around 4.7% predicted values dont match with true values.
In [171]:
confusion_matrix_sklearn(xgb_word2vec_tuned, X_valid_word2vec, y_valid_xgb_word2vec)
Observations:
- For validation dataset, we see that around 37% of the predicted values match the actual values.
In [172]:
# Predicting on train data
y_pred_train_xgb_word2vec_tuned = xgb_word2vec_tuned.predict(X_train_word2vec)
# Predicting on validation data
y_pred_valid_xgb_word2vec_tuned = xgb_word2vec_tuned.predict(X_valid_word2vec)
Classification Report
In [173]:
print(classification_report(y_train_xgb_word2vec, y_pred_train_xgb_word2vec_tuned))
Observations:
- For training data, We have a Recall score of 95%.
In [174]:
xgb_tuned_word2vec_report = classification_report(y_valid_xgb_word2vec, y_pred_valid_xgb_word2vec_tuned)
print(xgb_tuned_word2vec_report)
Observations:
- We got a weighted recall score of 37% which indicates that the model was overfitting.
- Negative sentiment recall score is 0%
- Neutral sentiment recall score is 76%
- Positive sentiment recall score is 0%
XGBoost (with hyperparametertuning and GloVe)¶
In [175]:
# Choose the type of classifier
xgb_tuned = XGBClassifier(random_state=42, eval_metric='mlogloss')
# defining the hyperparameter grid for tuning
parameters = {
"n_estimators": [10,30,50],
"subsample":[0.7,0.9,1],
"learning_rate":[0.05, 0.1,0.2],
"colsample_bytree":[0.7,0.9,1],
"colsample_bylevel":[0.5,0.7,1]
}
# defining the type of scoring used to compare parameter combinations
# we need to specify the mechanism of averaging as we have more than 2 target classes
scorer = metrics.make_scorer(metrics.recall_score, average='weighted')
# running the grid search
grid_obj = GridSearchCV(xgb_tuned, parameters, scoring=scorer, cv=3, n_jobs=-1)
grid_obj = grid_obj.fit(X_train_glove, y_train_xgb_glove)
In [176]:
# Creating a new model with the best combination of parameters
xgb_glove_tuned = grid_obj.best_estimator_
# Fit the new model to the data
xgb_glove_tuned.fit(X_train_glove, y_train_xgb_glove)
Out[176]:
Confusion Matrix
In [177]:
#Printing the confusion matrix
confusion_matrix_sklearn(xgb_glove_tuned, X_train_glove, y_train_xgb_glove)
Observations:
- We see that all predicted values matches the true/actual values and it looks like that the data is overfitting.
In [178]:
#Printing the confusion matrix
confusion_matrix_sklearn(xgb_glove_tuned, X_valid_glove, y_valid_xgb_glove)
Observations:
- For validation dataset, we see that around 46% of the predicted values match the actual values.
In [179]:
# Predicting on train data
y_pred_train_xgb_glove_tuned = xgb_glove_tuned.predict(X_train_glove)
# Predicting on validation data
y_pred_valid_xgb_glove_tuned = xgb_glove_tuned.predict(X_valid_glove)
Classification Report
In [180]:
print(classification_report(y_train_xgb_glove, y_pred_train_xgb_glove_tuned))
Observations:
- For training data, We have a Recall score of 100% which indicates that the model is overfitting.
In [181]:
xgb_tuned_glove_report = classification_report(y_valid_xgb_glove, y_pred_valid_xgb_glove_tuned)
print(xgb_tuned_glove_report)
Observations:
- We got a weighted recall score of 46% which indicates that the model was overfitting.
- Negative sentiment recall score is 30%
- Neutral sentiment recall score is 71%
- Positive sentiment recall score is 12%
XGBoost(with Hyperparameter Tuning and Sentence Transformer)¶
In [182]:
# Choose the type of classifier
xgb_tuned = XGBClassifier(random_state=42, eval_metric='mlogloss')
# defining the hyperparameter grid for tuning
parameters = {
"n_estimators": [10,30,50],
"subsample":[0.7,0.9,1],
"learning_rate":[0.05, 0.1,0.2],
"colsample_bytree":[0.7,0.9,1],
"colsample_bylevel":[0.5,0.7,1]
}
# defining the type of scoring used to compare parameter combinations
# we need to specify the mechanism of averaging as we have more than 2 target classes
scorer = metrics.make_scorer(metrics.recall_score, average='weighted')
# running the grid search
grid_obj = GridSearchCV(xgb_tuned, parameters, scoring=scorer, cv=3, n_jobs=-1)
grid_obj = grid_obj.fit(X_train_sent_transformer, y_train_xgb_sent_transformer)
In [183]:
# Creating a new model with the best combination of parameters
xgb_sent_tuned = grid_obj.best_estimator_
# Fit the new model to the data
xgb_sent_tuned.fit(X_train_sent_transformer, y_train_xgb_sent_transformer)
Out[183]:
In [184]:
#Printing the confusion matrix
confusion_matrix_sklearn(xgb_sent_tuned, X_train_sent_transformer, y_train_xgb_sent_transformer)
Observations:
- We see that all predicted values matches the true/actual values and it looks like that the data is overfitting.
In [185]:
#Printing the confusion matrix
confusion_matrix_sklearn(xgb_sent_tuned, X_valid_sent_transformer, y_valid_xgb_sent_transformer)
Observations:
- For validation dataset, we see that around 34% of the predicted values match the actual values.
In [186]:
# Predicting on train data
y_pred_train_xgb_sent_tuned = xgb_sent_tuned.predict(X_train_sent_transformer)
# Predicting on validation data
y_pred_valid_xgb_sent_tuned = xgb_sent_tuned.predict(X_valid_sent_transformer)
Classification Report
In [187]:
print(classification_report(y_train_xgb_sent_transformer, y_pred_train_xgb_sent_tuned))
Observations:
- For training data, We have a Recall score of 100% which indicates that the model is overfitting.
In [188]:
xgb_tuned_sent_report = classification_report(y_valid_xgb_sent_transformer, y_pred_valid_xgb_sent_tuned)
print(xgb_tuned_sent_report)
Observations:
- We got a weighted recall score of 34% which indicates that the model was overfitting.
- Negative sentiment recall score is 0%
- Neutral sentiment recall score is 65%
- Positive sentiment recall score is 12%
Models Performance Summary¶
In [189]:
# Summarize all the reports
def summarize_reports(reports, model_type):
"""Summarizes model reports in a structured format.
Args:
reports (dict): A dictionary of reports where keys are report names
and values are the actual report objects.
model_type (str): The type of model (e.g., "Random Forest", "XGBoost").
Returns:
None (prints the summary to the console)
"""
print(f"-----------------{model_type.upper()} MODELS---------------------- ")
for report_name, report in reports.items():
print(f"\n{report_name.replace('_', ' ').title()} Report:")
print(report)
print("-" * 50)
# Define dictionaries to hold your reports
random_forest_reports = {
"default_word2vec": default_word2vec_report,
"weighted_word2vec": weighted_word2vec_report,
"tuned_word2vec": tuned_word2vec_report,
"default_glove": default_glove_report,
"weighted_glove": weighted_glove_report,
"tuned_glove": tuned_glove_report,
"default_sentence_transformer": default_sent_report,
"weighted_sentence_transformer": weighted_sent_report,
"tuned_sentence_transformer": tuned_sent_report,
}
xgboost_reports = {
"default_word2vec": default_xgb_word2vec_report,
"tuned_word2vec": xgb_tuned_word2vec_report,
"default_glove": default_xgb_glove_report,
"tuned_glove": xgb_tuned_glove_report,
"default_sentence_transformer": default_xgb_sent_report,
"tuned_sentence_transformer": xgb_tuned_sent_report,
}
# Print the summarized reports
print("Metrics summary of all the models")
print("-" * 50)
summarize_reports(random_forest_reports, "Random Forest")
summarize_reports(xgboost_reports, "XGBoost")
Model Performance Summary :
- We have built 15 models( 9 Random Forest models and 6 XGBoost Model) using Word2vec, Glove and Sentence Transformer embedding. Our metric of interest is Recall.
We observe that there are top 4 models with the highest recall score of 46%. Below are the models
- Random Forest - Default Glove Model
- Random Forest - Weighted Sentence Transformer Model
- XGBoost - Tuned Glove Model
- XGBoost - Default Sentence Transformer Model
Lets dive further to look at other factors to choose the right model
Class-wise Recall :
- Random Forest - Default Glove Model
- Negative sentiment recall score is 20%
- Neutral sentiment recall score is 76%
- Positive sentiment recall score is 12%
- Random Forest - Weighted Sentence Transformer Model
- Negative sentiment recall score is 10%
- Neutral sentiment recall score is 82%
- Positive sentiment recall score is 12%
- XGBoost - Tuned Glove Model
- Negative sentiment recall score is 30%
- Neutral sentiment recall score is 71%
- Positive sentiment recall score is 12%
XGBoost - Default Sentence Transformer Model
- Negative sentiment recall score is 30%
- Neutral sentiment recall score is 59%
- Positive sentiment recall score is 38%
By Observation, we see XGBoost - Default Sentence Transformer Model class Wise recall score for Negative and positive sentiments are better than other models.
- Random Forest - Default Glove Model
Computational Efficiency :
- Default Models - need the least computational resources
- Weighted Models - Second least resources after the default Models
Hyper parameter tuning Models- needs more computational resources
For Sentence Transformers, we do not have to pre-process or clean the data, we can feed the input as is which saves computational resource
By Observation, we see that XGBoost - Default Sentence Transformer Model requires less computational resource
Other Metrics like F1-Score :
- The Second critical métricas is F1-Score which keeps a balance between Precision and Recall.
- Random Forest - Default Glove Model
- F1- Score is 40%
- Random Forest - Weighted Sentence Transformer Model
- F1-Score is 38%
- XGBoost - Tuned Glove Model
- F1-Score is 42%
XGBoost - Default Sentence Transformer Model
- F1-Score is 45%
By observation, we see that XGBoost - Default Sentence Transformer Model has the highest F1-Score of 45%
Final Model: Based on all the criteria above, XGBoost - Default Sentence Transformer Model is the best model.
Final Model Performance on Test Data¶
In [190]:
# Predicting on validation data
y_pred_test_xgb_sent_transformer = xgb_sent_transformer.predict(X_test_sent_transformer)
Confusion Matrix
In [192]:
#XGBoost doesnt take negative values/classes in y. So map the classes to positive number
y_test_xgb_sent_transformer = y_test_sent_transformer.map({-1: 0, 0: 1, 1: 2}) # Change -1 to 0, 0 to 1, and 1 to 2
In [193]:
confusion_matrix_sklearn(xgb_sent_transformer, X_test_sent_transformer, y_test_xgb_sent_transformer)
Classification Report
In [194]:
#included zero_division=1 to address the warning related to precision values
final_model_report = classification_report(y_test_xgb_sent_transformer, y_pred_test_xgb_sent_transformer, zero_division=1)
print(final_model_report)
Final Model Summary : (XGBoost - Default Sentence Transformer)
- Weighted Recall score on test data - 46%
- Class-wise Recall Scores
- Negative Sentiment - 40%
- Neutral Sentiment - 59%
- Positive Sentiment - 25%
- F1-Score
- F1-Score is 45%
Conclusion : Model has generalized well and has given the similar performance as validation dataset.
Weekly News Summarization¶
Important Note: It is recommended to run this section of the project independently from the previous sections in order to avoid runtime crashes due to RAM overload.
Installing and Importing the necessary libraries¶
In [1]:
!pip install git+https://github.com/abetlen/llama-cpp-python.git
In [ ]:
# Installation for GPU llama-cpp-python
# uncomment and run the following code in case GPU is being used
#!CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip install llama-cpp-python==0.1.85 -q
# Installation for CPU llama-cpp-python
# uncomment and run the following code in case GPU is not being used
#!CMAKE_ARGS="-DLLAMA_CUBLAS=off" FORCE_CMAKE=1 pip install llama-cpp-python -q
In [2]:
# Function to download the model from the Hugging Face model hub
from huggingface_hub import hf_hub_download
# Importing the Llama class from the llama_cpp module
from llama_cpp import Llama
# Importing the library for data manipulation
import pandas as pd
from tqdm import tqdm # For progress bar related functionalities
tqdm.pandas()
# to ignore unnecessary warnings
import warnings
warnings.filterwarnings("ignore")
Loading the data¶
In [3]:
# mounting Google Drive
from google.colab import drive
drive.mount('/content/drive')
In [4]:
# loading the dataset
df_summarization = pd.read_csv('/content/drive/My Drive/Colab Notebooks/NLP/Project/stock_news.csv')
In [5]:
data_summarization = df_summarization.copy()
Loading the model¶
In [26]:
import torch
from llama_cpp import Llama
# Check if CUDA is available
if torch.cuda.is_available():
device = torch.device('cuda')
else:
device = torch.device('cpu')
In [27]:
print(device)
In [6]:
model_name_or_path = "TheBloke/Mistral-7B-Instruct-v0.2-GGUF"
model_basename = "mistral-7b-instruct-v0.2.Q6_K.gguf"
In [7]:
# Using hf_hub_download to download a model from the Hugging Face model hub
# The repo_id parameter specifies the model name or path in the Hugging Face repository
# The filename parameter specifies the name of the file to download
model_path = hf_hub_download(
repo_id=model_name_or_path,
filename=model_basename
)
In [33]:
llm = Llama(
model_path=model_path,
n_threads=2, # CPU cores
n_batch=512, # Should be between 1 and n_ctx, consider the amount of VRAM in your GPU.
n_gpu_layers=43, # Change this value based on your model and your GPU VRAM pool.
n_ctx=5500, # Context window
)
Aggregating the data weekly¶
In [9]:
data_summarization["Date"] = pd.to_datetime(data_summarization['Date']) # Convert the 'Date' column to datetime format.
In [10]:
# Group the data by week using the 'Date' column.
weekly_grouped = data_summarization.groupby(pd.Grouper(key='Date', freq='W'))
In [11]:
weekly_grouped = weekly_grouped.agg(
{
'News': lambda x: ' || '.join(x) # Join the news values with ' || ' separator.
}
).reset_index()
print(weekly_grouped.shape)
In [12]:
weekly_grouped
Out[12]:
In [13]:
# creating a copy of the data
data_1 = weekly_grouped.copy()
Summarization¶
Note:
The model is expected to summarize the news from the week by identifying the top three positive and negative events that are most likely to impact the price of the stock.
As an output, the model is expected to return a JSON containing two keys, one for Positive Events and one for Negative Events.
For the project, we need to define the prompt to be fed to the LLM to help it understand the task to perform. The following should be the components of the prompt:
Role: Specifies the role the LLM will be taking up to perform the specified task, along with any specific details regarding the role
- Example:
You are an expert data analyst specializing in news content analysis.
- Example:
Task: Specifies the task to be performed and outlines what needs to be accomplished, clearly defining the objective
- Example:
Analyze the provided news headline and return the main topics contained within it.
- Example:
Instructions: Provides detailed guidelines on how to perform the task, which includes steps, rules, and criteria to ensure the task is executed correctly
- Example:
Instructions:
1. Read the news headline carefully.
2. Identify the main subjects or entities mentioned in the headline.
3. Determine the key events or actions described in the headline.
4. Extract relevant keywords that represent the topics.
5. List the topics in a concise manner.Output Format: Specifies the format in which the final response should be structured, ensuring consistency and clarity in the generated output
- Example:
Return the output in JSON format with keys as the topic number and values as the actual topic.
- Example:
Full Prompt Example:
You are an expert data analyst specializing in news content analysis.
Task: Analyze the provided news headline and return the main topics contained within it.
Instructions:
1. Read the news headline carefully.
2. Identify the main subjects or entities mentioned in the headline.
3. Determine the key events or actions described in the headline.
4. Extract relevant keywords that represent the topics.
5. List the topics in a concise manner.
Return the output in JSON format with keys as the topic number and values as the actual topic.Sample Output:
{"1": "Politics", "2": "Economy", "3": "Health" }
Note:
The model is expected to summarize the news from the week by identifying the top three positive and negative events that are most likely to impact the price of the stock.
As an output, the model is expected to return a JSON containing two keys, one for Positive Events and one for Negative Events.
For the project, we need to define the prompt to be fed to the LLM to help it understand the task to perform. The following should be the components of the prompt:
Role: Specifies the role the LLM will be taking up to perform the specified task, along with any specific details regarding the role
- Example:
You are an expert data analyst specializing in news content analysis.
- Example:
Task: Specifies the task to be performed and outlines what needs to be accomplished, clearly defining the objective
- Example:
Analyze the provided news headline and return the main topics contained within it.
- Example:
Instructions: Provides detailed guidelines on how to perform the task, which includes steps, rules, and criteria to ensure the task is executed correctly
- Example:
Instructions:
1. Read the news headline carefully.
2. Identify the main subjects or entities mentioned in the headline.
3. Determine the key events or actions described in the headline.
4. Extract relevant keywords that represent the topics.
5. List the topics in a concise manner.Output Format: Specifies the format in which the final response should be structured, ensuring consistency and clarity in the generated output
- Example:
Return the output in JSON format with keys as the topic number and values as the actual topic.
- Example:
Full Prompt Example:
You are an expert data analyst specializing in news content analysis.
Task: Analyze the provided news headline and return the main topics contained within it.
Instructions:
1. Read the news headline carefully.
2. Identify the main subjects or entities mentioned in the headline.
3. Determine the key events or actions described in the headline.
4. Extract relevant keywords that represent the topics.
5. List the topics in a concise manner.
Return the output in JSON format with keys as the topic number and values as the actual topic.Sample Output:
{"1": "Politics", "2": "Economy", "3": "Health" }
Utility Functions¶
In [14]:
# defining a function to parse the JSON output from the model
def extract_json_data(json_str):
import json
try:
# Find the indices of the opening and closing curly braces
json_start = json_str.find('{')
json_end = json_str.rfind('}')
if json_start != -1 and json_end != -1:
extracted_category = json_str[json_start:json_end + 1] # Extract the JSON object
data_dict = json.loads(extracted_category)
return data_dict
else:
print(f"Warning: JSON object not found in response: {json_str}")
return {}
except json.JSONDecodeError as e:
print(f"Error parsing JSON: {e}")
return {}
In [17]:
import nltk
# Download the 'punkt_tab' resource
nltk.download('punkt_tab')
from nltk.tokenize import word_tokenize
token_counts = [len(word_tokenize(text)) for text in data_1['News']]
max_tokens = max(token_counts)
print(max_tokens)
Defining the response function¶
In [45]:
Instruction_1 = """
Role: You are an expert financial analyst specializing in market sentiment analysis and news summarization. Your primary role is to analyze weekly news articles related to a specific company and determine the top three positive and negative events that are most likely to affect its stock price.
Task: Analyze the provided news articles for the past week and identify the top three positive and negative events. These events should be the most significant occurrences reported in the news that could potentially influence the company's stock price. Summarize these events concisely and objectively.
Instructions:
1. Carefully read each news article provided for the specified week.
2. Extract key events or topics discussed in the articles.
3. Categorize the events as positive or negative based on their potential impact on the company's stock price. For example, a new product launch would generally be considered a positive event, while a product recall would be considered a negative event.
4. Rank the positive and negative events based on their significance and potential impact.
5. Select the top three most impactful positive events and the top three most impactful negative events.
6. Summarize each selected event in a clear and concise manner, avoiding subjective opinions or interpretations. Focus on factual reporting and avoid speculation.
7.Present the summarized events in a JSON format with two keys: "Positive Events" and "Negative Events." Each key should contain a list of the three summarized events in order of impact.
Example JSON Output:
{
"Positive Events": [
"Company announced a strategic partnership with a major industry player, potentially expanding its market reach.",
"Positive earnings report exceeding analysts' expectations, indicating strong financial performance.",
"New product launch receiving positive reviews and generating significant customer interest."
],
"Negative Events": [
"Product recall due to safety concerns, impacting sales and brand reputation.",
"Regulatory investigation initiated against the company, potentially leading to fines or penalties.",
"Key executive unexpectedly resigned, raising concerns about leadership stability."
]
}
"""
In [46]:
#length of the instructions
len(Instruction_1)
Out[46]:
In [47]:
#Defining the response function
def response_mistral(prompt, news):
model_output = llm(
f"""
[INST]
{prompt}
News Articles: {news}
[/INST]
""",
max_tokens=5500, #Complete the code to set the maximum number of tokens the model should generate for this task.
temperature=0, #Complete the code to set the value for temperature.
top_p=0.95, #Complete the code to set the value for top_p
top_k=50, #Complete the code to set the value for top_k
echo=False,
)
final_output = model_output["choices"][0]["text"]
return final_output
Checking the model output on a sample¶
Note: Use this section to test out the prompt with one instance before using it for the entire weekly data.
In [48]:
#prompt with one instance
#use iloc to get one week
#create dataframe to store the output
test_data = response_mistral(Instruction_1, data_1['News'][0])
In [36]:
print(test_data)
In [49]:
import pandas as pd
from IPython.display import display
# Assuming data_1 is your DataFrame and 'News' is the column containing news articles
# Set display options to show all text
pd.set_option("display.max_colwidth", None) # Display full column width
In [50]:
# Display the entire text of the first news article without truncation
display(data_1['News'][0])
Checking the model output on the weekly data¶
In [51]:
data_1['model_response'] = data_1['News'].progress_apply(lambda x: response_mistral(Instruction_1, x))
Formatting the model output¶
Extract the JSON data
In [52]:
data_1['model_response_parsed'] = data_1['model_response'].apply(extract_json_data)
In [53]:
data_1['model_response_parsed']
Out[53]:
Checking for empty dictionary
In [54]:
data_1[data_1["model_response_parsed"]=={}]
Out[54]:
Example model parsed response
In [55]:
data_1['model_response_parsed'][4]
Out[55]:
Create a dataframe from the JSON data
In [76]:
model_response_parsed_df = pd.json_normalize(data_1['model_response_parsed'])
model_response_parsed_df.head(2)
Out[76]:
In [70]:
data_with_parsed_model_output = pd.concat([data_1, model_response_parsed_df], axis=1)
Remove square brackets from Positive and Negative Events
In [71]:
import re
def remove_brackets(text):
if isinstance(text, list): # Check if text is a list
text = ', '.join(text) if text else '' # Join list elements into a string
return re.sub(r'\[.*?\]', '', str(text)) # Convert text to string before applying re.sub
In [72]:
data_with_parsed_model_output['Positive Events'] = data_with_parsed_model_output['Positive Events'].apply(remove_brackets)
data_with_parsed_model_output['Negative Events'] = data_with_parsed_model_output['Negative Events'].apply(remove_brackets)
In [73]:
data_with_parsed_model_output.head(2)
Out[73]:
In [74]:
#remove model_response and model_response_parsed
final_summary = data_with_parsed_model_output.drop(['model_response', 'model_response_parsed'], axis=1)
Display the Final Summary[ Date, News, Positive Events and Negative Events summary]
In [77]:
final_summary.head(2)
Out[77]:
Conclusions and Recommendations¶
Conclusion and Business Recommendation on Sentimental Analysis:
- We see that the dataset is imbalanced dataset. We could improve on the data by applying undersampling or oversampling of data, collect more data with negative and positive sentiments to take care of the imbalance issue.
- As per EDA, The stock price trend
- begins to rise up from begining of the month, peaks before the mid of the month, the drops low by mid of the month.It stays low from mid of the month to the beginning of the next month and the cycle continues.
- The above cycle is true for all stock prices - Open, High, Low and Close price.
- We see that the Volume traded drops drastically from Month 1 to Month 2 and then remains low. We recommend the business to look into the reasons as why the stock volume decretase.
- We built ML models - Random Forest and XGBoost Models . We were only able to get a recall of 46%. We recommend to look at other model and pre-trained models to increase the recall.
- By predicting the sentiments, we will be able to predict how the stock price is going up or down.
Conclusion and Business Recommendation on weekly news summarization:
- Use other pretrained models and compare the model outputs
- It takes Lot of computacional resource to get summarization of positive and negative events on the Weekly data. Recommend Business to invest more in getting higher computational resources.
- This will enable experimenting with different prompts to get better model response.
- This will entable experimenting with different model parameters like max_token, top_p, temperature etc.
- By providing the Summary of 3 positive events and 3 negative events, we will be able to use it as one of the parameters to predict how the stock is going to perform in the Market.
- Business can look at the news articles and sentiments and take appropriate action to mitigate the risks associated with negative sentiments, use positive sentiments as marketing strategy.
-
Power Ahead